Sanitize auto-generated output tool name to support generic types #2979

lionpeloux · 2025-09-22T10:33:31Z

We ensure that auto-generated tool ouput name are properly sanitazed so they conform to [a-z0-9-_].
All char not in this pattern will simply be ignored/skipped in the final name.

The error was first remarked when using generic classes as output_type, where brakets in the tool name would be rejected by the provider API.

DouweM · 2025-09-26T21:56:56Z

@lionpeloux Thanks Lionel! Can you please add a test?

github-actions · 2025-10-04T14:00:39Z

This PR is stale, and will be closed in 3 days if no reply is received.

lionpeloux · 2025-10-04T18:28:03Z

I’ll get back to it next week hopefully.

github-actions · 2025-10-12T14:00:42Z

This PR is stale, and will be closed in 3 days if no reply is received.

lionpeloux · 2025-10-14T08:08:06Z

Fix Applied

I've identified and fixed the bug in the tool name sanitization regex.

The Issue

The original regex pattern [^a-z0-9-_] was only allowing lowercase letters, which caused it to remove uppercase letters from class names. For example:

Foo → oo (the F was stripped)
Bar → ar (the B was stripped)

This caused all the test failures you were seeing.

The Fix

Changed the regex pattern to [^a-zA-Z0-9-_] to preserve both uppercase and lowercase letters while still removing invalid characters like brackets from generic type names.

Testing

✅ All previously failing tests now pass
✅ Added a new test case test_output_type_generic_class_name_sanitization to verify that generic class names with brackets are properly sanitized
✅ The fix correctly handles the original issue from Automatic tool output name generation fails with Generic types containing brackets #2929 where Result[StringData] gets sanitized to ResultStringData

The PR should now pass all CI checks!

Co-authored-by: Copilot <[email protected]>

…tput.py` (pydantic#2991)

…#3025)

…ctured output streaming (pydantic#3032)

…ndor_metadata` (pydantic#2987) Co-authored-by: Douwe Maan <[email protected]>

…antic#2957)

pydantic#3082)

…t_stream_handler=...)` (pydantic#3084)

The previous regex pattern `[^a-z0-9-_]` was removing uppercase letters from class names, causing test failures. For example, "Foo" became "oo". This fix changes the pattern to `[^a-zA-Z0-9-_]` to preserve both uppercase and lowercase letters while still removing invalid characters like brackets from generic type names (e.g., `Result[StringData]`). Also adds a test case to verify that generic class names with brackets are properly sanitized while preserving valid characters. Fixes test failures in test_response_multiple_return_tools and related tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

lionpeloux · 2025-10-14T08:13:01Z

PR Branch Updated with Latest Main

I've successfully rebased the PR branch onto the latest main branch. The branch is now up to date with all recent changes, including:

48 commits from main since the original PR was created
Major updates including image generation support, OpenTelemetry instrumentation v3, MCP improvements, and more

What Changed

The PR now includes:

✅ The original tool name sanitization feature
✅ The bug fix for uppercase letter preservation in class names
✅ A comprehensive test case for generic type sanitization
✅ All the latest changes from main

Testing Status

All tests pass locally after the rebase, confirming that the fix is compatible with the latest codebase changes.

The CI should now run against the updated branch and all tests should pass! 🚀

Resolved conflicts by: - Accepting all incoming changes from main - Preserving the tool name sanitization fix in _output.py - Re-adding the test_output_type_generic_class_name_sanitization test The PR now includes: - Tool name sanitization with uppercase letter support [^a-zA-Z0-9-_] - Comprehensive test for generic type bracket removal - All latest changes from main (Prefect support, image generation, etc.)

lionpeloux · 2025-10-14T09:53:13Z

@DouweM I've added the test. Let me know it something goes wrong.

DouweM · 2025-10-14T10:52:47Z

tests/test_agent.py

+    assert len(m.last_model_request_parameters.output_tools) == 2
+
+    # Check that tool names don't contain brackets
+    tool_names = [tool.name for tool in m.last_model_request_parameters.output_tools]
+    for tool_name in tool_names:
+        assert '[' not in tool_name, f"Tool name '{tool_name}' contains brackets"
+        assert ']' not in tool_name, f"Tool name '{tool_name}' contains brackets"
+        # Verify the name follows the pattern [a-zA-Z0-9_-]
+        assert re.match(r'^[a-zA-Z0-9_-]+$', tool_name), f"Tool name '{tool_name}' contains invalid characters"


I'd like to add this (the snapshot will be filled in automatically when you run the test so that we see the names we ended up with.

Suggested change

assert len(m.last_model_request_parameters.output_tools) == 2

# Check that tool names don't contain brackets

tool_names = [tool.name for tool in m.last_model_request_parameters.output_tools]

for tool_name in tool_names:

assert '[' not in tool_name, f"Tool name '{tool_name}' contains brackets"

assert ']' not in tool_name, f"Tool name '{tool_name}' contains brackets"

# Verify the name follows the pattern [a-zA-Z0-9_-]

assert re.match(r'^[a-zA-Z0-9_-]+$', tool_name), f"Tool name '{tool_name}' contains invalid characters"

tool_names = [tool.name for tool in m.last_model_request_parameters.output_tools]

assert tool_names == snapshot()

github-actions · 2025-10-22T14:01:02Z

This PR is stale, and will be closed in 3 days if no reply is received.

From pydantic#2929 We ensure that auto-generated tool output names are properly sanitized so they conform to `[a-zA-Z0-9-_]`. All characters not in this pattern will simply be ignored/skipped in the final name. The error was first remarked when using generic classes as `output_type`, where brackets in the tool name would be rejected by the provider API. Added snapshot test to verify the sanitized tool names are generated correctly from generic types like Result[StringData] and Result[int]. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

lionpeloux · 2025-10-23T05:13:31Z

Hi @DouweM, I've merged your suggestion. Thanks for your review.

DouweM · 2025-10-23T14:06:39Z

tests/test_agent.py


+def test_output_type_generic_class_name_sanitization(create_module: Callable[[str], Any]):
+    """Test that generic class names with brackets are properly sanitized."""
+    module_code = '''


We can inline this as regular code -- we're only using a stringified module above because we're injecting code {union_code}.

@DouweM

As suggested by @DouweM, the test now defines classes directly inline instead of using the create_module helper with a string, since we're not dynamically generating code. Note: The snapshot changed because inline classes have their function scope in the qualified name, resulting in a longer sanitized tool name. This still correctly tests bracket sanitization. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

lionpeloux · 2025-10-23T14:33:10Z

@DouweM I've inlined the code as you suggested.

Note about the snapshot change: The tool name changed from final_result_ResultStringData to final_result_Resulttest_output_type_generic_class_name_sanitizationlocalsStringData because when classes are defined inline within a function, their fully qualified name includes the function scope (e.g., test_output_type_generic_class_name_sanitization.<locals>.StringData).

The sanitizer processes this qualified name and strips out the dots and special characters, resulting in the longer name. This is correct behavior and still properly tests that brackets from generic types like Result[StringData] are sanitized.

Let me know if you'd prefer to keep the create_module approach to have cleaner tool names in the snapshot, or if this is acceptable.

DouweM · 2025-10-23T14:47:23Z

@lionpeloux Ah yeah that's a bit confusing. I'm OK with defining the classes at the top level of the file, I'd rather have that than a string module.

lionpeloux · 2025-10-24T14:24:33Z

@DouweM Done! I've moved the classes to module level as you suggested.

Changes:

Added Generic and TypeVar to the typing imports
Defined ResultGeneric and StringData classes at module level (similar to the Person class pattern elsewhere in the file)
Simplified the test function to use these module-level classes
Tool names are now clean: final_result_ResultGenericStringData and final_result_ResultGenericint

This gives us the best of both worlds - readable inline code (no string modules) and clean snapshot names (no function scope pollution). Test passes successfully!

DouweM · 2025-10-24T15:06:15Z

@lionpeloux Don't forget to push :)

@DouweM

As suggested by @DouweM, moved ResultGeneric and StringData classes to the top of the test file (after Person class) instead of defining them inline within the test function. This approach: - Avoids the string module approach (create_module) - Provides clean tool names in snapshots without function scope - Follows the existing pattern in the test file Tool names are now clean: final_result_ResultGenericStringData and final_result_ResultGenericint. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

lionpeloux · 2025-10-24T19:03:11Z

@DouweM Classes are now at the TOP of the file!

I moved ResultGeneric and StringData to line 100-112 (right after the Person class), with a descriptive comment explaining their purpose for testing tool name sanitization with generic types.

The test passes successfully with clean tool names in the snapshot.

DouweM · 2025-10-24T19:32:49Z

@lionpeloux Thanks Lionel!

add sanitization to auto-generated output tool name

a5ae245

lionpeloux changed the title ~~add sanitization to auto-generated output tool name~~ Add sanitization to auto-generated output tool name Sep 22, 2025

lionpeloux marked this pull request as ready for review September 22, 2025 10:40

DouweM self-assigned this Sep 26, 2025

DouweM added the awaiting author revision label Sep 26, 2025

github-actions bot added the Stale label Oct 4, 2025

github-actions bot removed the Stale label Oct 5, 2025

github-actions bot added the Stale label Oct 12, 2025

DouweM removed the Stale label Oct 13, 2025

Kludex and others added 16 commits October 14, 2025 10:11

Update pyproject.toml to be PEP639 compliant (pydantic#3001)

6322b70

Fix OpenAI docs code example (pydantic#3003)

e1784e6

Add operation.cost metric to instrumented models (pydantic#3013)

60dd3de

Co-authored-by: Copilot <[email protected]>

docs: replace "customisation" by "customization" (pydantic#3019)

86bc16e

move OutputObjectDefinition from private _output.py to public `ou…

a19a96d

…tput.py` (pydantic#2991)

Raise error when using Anthropic thinking with output tools (pydantic…

edf4d2d

…#3025)

Bump temporalio to 1.18.0 (pydantic#3027)

79a889c

Bump genai-prices to 0.0.28 (pydantic#3030)

329fc47

Document that Gemini requires native or prompted output mode for stru…

2f77d3d

…ctured output streaming (pydantic#3032)

Update Ollama docs instructions (pydantic#2993)

c7fdf01

Fix streaming gpt-oss using Ollama (pydantic#3035)

9804c42

Add claude-sonnet-4-5 to known model names (pydantic#3033)

d0d28c3

Update docs and tests for DBOS v2.0 (pydantic#3004)

e803828

Support callable classes as history processors (pydantic#2988)

f124391

Support OpenAI image detail on ImageUrl and BinaryContent via `ve…

e92d59b

…ndor_metadata` (pydantic#2987) Co-authored-by: Douwe Maan <[email protected]>

Expose .messages, .toolsets types in top-level pydantic_ai (pyd…

774e266

…antic#2957)

DouweM and others added 4 commits October 14, 2025 10:12

Add content (e.g. files) returned by tool to FunctionToolResultEvent (

6c02bc9

pydantic#3082)

Add Agent.run_stream_events() convenience method wrapping `run(even…

1b974ea

…t_stream_handler=...)` (pydantic#3084)

Change 'Join Slack' link to direct link (pydantic#3085)

a53d87a

lionpeloux force-pushed the pr-tool-output-naming branch from 74d15fb to b445f0b Compare October 14, 2025 08:12

DouweM requested changes Oct 14, 2025

View reviewed changes

github-actions bot added the Stale label Oct 22, 2025

lionpeloux force-pushed the pr-tool-output-naming branch from 30e9921 to a854960 Compare October 23, 2025 04:37

Merge remote-tracking branch 'upstream/main' into pr-tool-output-naming

f16acbc

github-actions bot removed the Stale label Oct 23, 2025

DouweM reviewed Oct 23, 2025

View reviewed changes

DouweM changed the title ~~Add sanitization to auto-generated output tool name~~ Sanitize auto-generated output tool name to support generic types Oct 23, 2025

Merge remote-tracking branch 'upstream/main' into pr-tool-output-naming

2a7cdb5

lionpeloux requested a review from DouweM October 23, 2025 14:37

Merge remote-tracking branch 'upstream/main' into pr-tool-output-naming

a8d8c78

DouweM merged commit efa1e26 into pydantic:main Oct 24, 2025
31 checks passed

Sanitize auto-generated output tool name to support generic types #2979

Sanitize auto-generated output tool name to support generic types #2979

Uh oh!

Conversation

lionpeloux commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DouweM commented Sep 26, 2025

Uh oh!

github-actions bot commented Oct 4, 2025

Uh oh!

lionpeloux commented Oct 4, 2025

Uh oh!

github-actions bot commented Oct 12, 2025

Uh oh!

lionpeloux commented Oct 14, 2025

Fix Applied

The Issue

The Fix

Testing

Uh oh!

lionpeloux commented Oct 14, 2025

PR Branch Updated with Latest Main

What Changed

Testing Status

Uh oh!

lionpeloux commented Oct 14, 2025

Uh oh!

DouweM Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 22, 2025

Uh oh!

lionpeloux commented Oct 23, 2025

Uh oh!

DouweM Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

lionpeloux commented Oct 23, 2025

Uh oh!

DouweM commented Oct 23, 2025

Uh oh!

lionpeloux commented Oct 24, 2025

Uh oh!

DouweM commented Oct 24, 2025

Uh oh!

lionpeloux commented Oct 24, 2025

Uh oh!

Uh oh!

DouweM commented Oct 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants

lionpeloux commented Sep 22, 2025 •

edited

Loading